Repository navigation
Conversation
The HTML Processor understands HTML regardless of how it's written, but many other functions are unable to do so. There are all sorts of syntax peculiarities and semantics that would be helpful to eliminate using the knowledge contained in the HTML Processor. This patch introduces `WP_HTML_Processor::normalize( $html )` as a method which takes a fragment of HTML as input and then returns a serialized version of the input, "cleaning it up" by balancing all tags, providing all missing optional tags, re-encoding all text, removing all duplicate attributes, and double-quote-escaping all attribute values. Core-62036
|
The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the Core Committers: Use this line as a base for the props when committing in SVN: To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook. |
commented
Sep 11, 2024
Test using WordPress PlaygroundThe changes in this pull request can previewed and tested using a WordPress Playground instance. WordPress Playground is an experimental project that creates a full WordPress instance entirely within the browser. Some things to be aware of
For more details about these limitations and more, check out the Limitations page in the WordPress Playground documentation. |
| */ | ||
| public function serialize(): ?string { | ||
| if ( WP_HTML_Tag_Processor::STATE_READY !== $this->parser_state ) { | ||
| return null; |
There was a problem hiding this comment.
Would this make sense to throw an exception?
There was a problem hiding this comment.
I've tried to avoid throwing exceptions in use code. Tell me more about the value of potentially crashing vs. returning null
There was a problem hiding this comment.
Well, I guess to give more information about why it is returning null. Maybe _doing_it_wrong() then would be better? It would be helpful to get feedback in code for what is documented:
* This differs from {@see WP_HTML_Processor::normalize} in that it starts with
* a specific HTML Processor, which _must_ not have already started scanning;
* it must be in the initial ready state and will be in the completed state once
* serialization is complete.
There was a problem hiding this comment.
ah okay I see now. another thought I had was resetting to the beginning, parsing, and returning to the previous location, which involves double-parsing if already mid-way through a document.
There was a problem hiding this comment.
I've called wp_trigger_error() in these cases.
| while ( $this->next_token() ) { | ||
| $token_type = $this->get_token_type(); | ||
|
|
||
| switch ( $token_type ) { |
There was a problem hiding this comment.
There was a problem hiding this comment.
ah good catch: it should also serialize the PI node tag name, which would match what you wrote. looks like this needs a review of all of the invalid comment syntax
There was a problem hiding this comment.
these should all be updated now. if something lingers I'd like to fix it, but ultimately if we botch an invalid comment, I'm guessing it's not the end of the world.
these will go into test cases.
| } | ||
|
|
||
| if ( ! $in_html && $this->has_self_closing_flag() ) { | ||
| $html .= '/'; |
There was a problem hiding this comment.
While not required, it seems a space is usually added here in the wild, right? (e.g. Prettier does this)
| $html .= '/'; | |
| $html .= ' /'; |
There was a problem hiding this comment.
a good thought. with double-quoted attributes it's not relevant, but with unquoted attribute values it becomes relevant. we don't need that since we control quoting. maybe it's best to add it in anyway for the same of other tools.
| if ( null !== $this->get_last_error() ) { | ||
| return null; | ||
| } |
There was a problem hiding this comment.
Similarly here, it would be helpful to know why it returned null.
There was a problem hiding this comment.
this is a tougher question because it would conflate the basic ?string return value. practically I think this can only occur if the HTML is unsupported (in which case we really shouldn't return any processed string) or we've run out of bookmarks (which should be unrealistically rare - and that reminds me, I found 2500 bookmarks sufficient to parse everything in my set of ~300k HTML documents, and I intend on upping the default value to support that for 6.7).
suppose you know why this failed: what would you do in response?
There was a problem hiding this comment.
It could also use _doing_it_wrong() here too to communicate that information, I suppose. Or rather, wp_trigger_error() would be the more relevant function. If I knew why it failed then I wouldn't have to figure out why it failed. True it probably wouldn't impact the result in the end, but for debugging it would be useful.
Looking at where last_error is set, it seems to always coincide with throwing an exception anyway. So in practice would this if statement ever be entered?
There was a problem hiding this comment.
the exceptions thrown internally are caught and shut down parsing, but does not crash. unsupported content exceptions are returned via get_unsupported_exception()
There was a problem hiding this comment.
I've called wp_trigger_error() in these cases.
Co-authored-by: Weston Ruter <westonruter@git.wordpress.org>
If code later in the processing pipeline adds unquoted attributes and doesn't add the requisite space following that, then another parser might find that the solidus is part of the attribute value instead of serving as a self-closing flag. Co-authored-by: Weston Ruter <westonruter@git.wordpress.org>
Co-authored-by: Weston Ruter <westonruter@git.wordpress.org>
Co-authored-by: Weston Ruter <westonruter@git.wordpress.org>
Co-authored-by: Weston Ruter <westonruter@git.wordpress.org>
left a comment
There was a problem hiding this comment.
This is pretty exciting, I'd like to start adding tests for it.
I just added it to the html api debugger when supported.
I'd love to start adding tests for this. One good test will be idempotency, where after an initial normalization, subsequent normalizations will be identical.
This mentions null bytes here specifically:
Text will be re-encoded, null bytes handled, and invalid UTF-8 replaced with U+FFFD.
I think that's working correctly in text. Should it also be handled in tag names, attribute names, and attribute values?
Input (null bytes replaces for clarity)
<div␀-nb nb-att-␀-="nb-val-␀-">Normalized output:
<div␀-nb nb-att-␀-="nb-val-␀-"></div␀-nb>Expected:
<div�-nb nb-att-�-="nb-val-�-"></div�-nb>|
There are some known issues from HTML5lib tests similar to the PI problems mentioned here: #7331 (comment) There's no good way to read the comment under some circumstances and something like a Each of these does not satisfy the PI constraint (missing There are a couple of cases like this, I think they're all CORRECTION: I've edited it, it was initially incorrect. The bogus comments starting with
|
commented
Sep 12, 2024
|
We'd talked about a method to really inspect different types of comment text content. I've proposed a method in #7342. That would be helpful here. |
c5f1924 to
92558b9
Compare

Trac ticket: Core-62036.
See also dmsnell#20
See also dmsnell#21
The HTML Processor understands HTML regardless of how it's written, but
many other functions are unable to do so. There are all sorts of syntax
peculiarities and semantics that would be helpful to eliminate using the
knowledge contained in the HTML Processor.
This patch introduces
WP_HTML_Processor::normalize( $html )as amethod which takes a fragment of HTML as input and then returns a
serialized version of the input, "cleaning it up" by balancing all
tags, providing all missing optional tags, re-encoding all text,
removing all duplicate attributes, and double-quote-escaping all
attribute values.